Homework II: SVD analysis & Life Tables

1 Objectives

This notebook aims at

• working with tables (data.frames, tibbles, data.tables, …) using dplyr or any other query language (as provided for example by data.table)
• visualizing demographic data as provided by Human Mortality Database organization.
• using PCA and other matrix oriented methods (CCA) to explore a multivariate datasets (lifetables may be considered as multivariate datasets)

2 Life tables data (ETL)

Life data tables are downloaded from https://www.mortality.org.

See also https://www.lifetable.de.

We investigate life tables describing countries from Western Europe (France, Great Britain –actually England and Wales–, Italy, the Netherlands, Spain, and Sweden) and the United States.

We load the one-year lifetables for female, male and whole population for the different countries.

2.1 Retypage

We begin by retyping the columns. The schema of df, our universal table, is the following :

Column Name	Column Type
Year	integer
Age	integer
mx	double
qx	double
ax	double
lx	integer
dx	integer
Lx	integer
Tx	integer
ex	double
Country	factor
Gender	factor

In order to avoid coercion problems and NA warnings, the column Age : 110+, is processed as Age : 110.

2.2 Western countries in 1948

Here we plot the central death rates of all Countries at all ages for the year 1948.

plot

Spain and Italy have higher death rates than the other countries from 0 to 30-45.

We can see that the lower central death rates are between 1 and 15 years old. Between 15 and 30 years old, this rate seems to increase a bit, and past 30 years old, the graph follows a linear function, meaning that the probability of dying past 30 years old become almost proportional to the age.

We also notice that the women have lower death rates than men.

We can see that between 0 and 45 years old France, Spain and Italy have higher central death rates than USA, while the others countries have similar or lower death rates. Infant mortality peaks is 5 times more important in Italy and Spain than in the US. However, in Sweden and Netherlands the death_rates are lower than in the US, especially for men between 30 and 60 years old, the mortality is almost half of the american one. Between 45 and 75 years old european countries have lower death rates than america. Past 75 years old, the disparities sooth, except maybe for Spanish women who have a minor mortality.

2.3 Death rates evolution since WWII

Now we will take a look at the evolution of death rates after the second world war. We plot the central death rates as a function of Age for the years 1946, 1956, 1966, … , 2016.

We can see that the mortality quotients have decreased decade by decade, especially at birth, and are generally at their lowest for the year 2016. We observe the same phenomenon during all these years than in 1948, the central death rates are low from 1 to 15, then increase and reach a peak around 25 years old, and finally seems to increase proportionaly from 30 years old. We also see that women have lower death rates than men, for all countries.

The peak around 25 years old is really noticeable for the men in all countries, while for women it isn’t as visible. This may be cause by suicide, because while women have an higher attempted suicide risks, men tend to employ more radical methods to kill themselfs. In 2008 the male-female rate ratio of suicide in Europe was 4.0 and in the USA 3.6, according to Suicide in the World, by Peeter Värnik. An other factor might be the higher propotion of men dying in motor vehicle crashes, for example in 2018 71% of motor vehicule deaths were men in the US. Men tend to have more high risks driving practices than women, in 2008 40% of males drivers killed on the road had BACs ≥ 0.08 percent, against 20% for females. All of these number come from this analysis of data from the U.S. Department of Transportation’s Fatality Analysis Reporting System (FARS).

This spike appears in 1956 and 1966, making the male’s curve diverge from the women one, but tend to flatten in 2006 and 2016 except in the US.

We observe that the mortality quotients for USA are higher than the european ones.

Here the function ratio_mortality_rateswith signaturefunction(df, reference_year=1946, target_years=seq(1946, 2016, 10))` that takes as input :

• a dataframe with the same schema as life_table,
• a reference year ref_year and
• a sequence of years target_years

and that returns a dataframe with schema:

Column Name	Column Type
Year	integer
Age	integer
mx	double
mx.ref_year	double
Country	factor
Gender	factor

where (Country, Year, Age, Gender) serves as a primary key, mx denotes the central death rate at Age for Year and Gender in Country whereas mx_ref_year denotes central death rate at Age for argument reference_year in Country for Gender.

We now draw plots displaying the ratio \(m_{x,t}/m_{x, 1946}\) for ages \(x \in 1, \ldots, 90\) and year \(t\) for \(t \in 1956, \ldots, 2016\) where \(m_{x,t}\) is the central death rate at age \(x\) during year \(t\).

We observe that the central death rates has decreased for every country and every age since 1946. The infant mortality, between 0 and 15 years old, in 2016 is 10 times lower than in 1946 in European countries. It appears that the mortality of the women has decreased the most since 1946.

In the US this drop is less drastic, and we can notice that around 30 for males for 2006 and 2016, the ratio doesn’t follow the overall diminuition unlike in the previous decade. There is a similar phenomenon for females, but less important.

3 Trends

On these plots, we can see that the infant mortality has decreased as the time gone by, particularly at birth and at 1 year old. This diminuition has speed up since 1910 - 1930 depending of the country, and we might link this to the drastic change in childbirth, and global hygiene during this period.

Once again we observe the decreasing of the mortality quotients as a function of time. For European countries, we notice 2 peaks more importants for men than female (aged 15, 20 and 40 years old) corresponding to WWI and WWII (and spanish civil war for Spain), except for Sweden which remaind neutral during the second conflict. There is also a little peak for the USA male at 20 during WWII. For France we can see others peaks for males and females after 1850 during the Franco-Prussian War, and other conflicts in this period.

We also observe a major drop for women at 15 and 20 years old from all countries after WWII, which might be linked, as for the drop of infant mortality, to the progress made with general hygiene, delivery, and the end of barbar method such as the Twilight Sleep.

4 Rearrangement

Now we compute a new dataframe df_pivot, with primary key (Gender,Year,Country), with a column for each Age from 0 to 110.

According to https://www.lifeexpectancy.org/lifetable.shtml, life expanctancy \(e(x)\) of persons alive at age \(x\) is computed with the following method :

\[ e(x) = \frac {T(x)} {l(x)} \] where

\[ l(x + 1) = l(x) \cdot e^{-m(x)} \] So we have \[ l(x + 2) = l(x + 1) \cdot e^{-m(x + 1)} = l(x) \cdot e^{-m(x)} \cdot e^{-m(x + 1)} = l(x) \cdot e^{-(m(x) + m(x + 1))} \] Therefore, \(\forall n \in [|1, 110 - x|]\):

\[ l(x + n) = l(x) \cdot e^{-\sum_{i = 0}^{n - 1} m(x + i)} \] And for \(T(x)\): \[ T(x) = \sum_{n = 0}^{110 - x} L(x + n)\\ L(x) = \frac 1 2 (l(x) + l(x + 1)) = \frac 1 2 l(x) (1 + e^{-m(x)}) \] So \(\forall n \in [|1, 110 - x|]\) \[ L(x + n) = \frac 1 2 l(x + n) (1 + e^{-m(x + n)})\\ = \frac 1 2 l(x) \cdot e^{-\sum_{i = 0}^{n - 1} m(x + i)} (1 + e^{-m(x + n)}) \] Which gives us: \[ T(x) = \sum_{n = 0}^{110 - x} L(x + n) = \frac 1 2 l(x) (1 + e^{-m(x)}) + \sum_{n = 1}^{110 - x} \frac 1 2 l(x) \cdot e^{-\sum_{i = 0}^{n - 1} m(x + i)} (1 + e^{-m(x + n)})\\ = \frac 1 2 l(x) \cdot [(1 + e^{-m(x)}) + \sum_{n = 1}^{110 - x} e^{-\sum_{i = 0}^{n - 1} m(x + i)} \cdot (1 + e^{-m(x + n)})] \] This way:

\[ e(x) = \frac {T(x)} {l(x)} = \frac 1 2 \cdot [(1 + e^{-m(x)}) + \sum_{n = 1}^{110 - x} e^{-\sum_{i = 0}^{n - 1} m(x + i)} \cdot (1 + e^{-m(x + n)})] \]

- [ ] Using life_table_pivot compute life expectancy at birth for each Country, Gender and Year

#Life expectancy

life_expectancy is a function that takes as input a vector of mortality quotients, as well as an age, and returns the residual life expectancy corresponding to the vector and the given age.

Then we have life_expectancy_all, a function that takes as input a dataframe with the same schema as life_table and returns a data frame with columns Country, Gender, Year, Age defining a primary key and a column res_lex containing residual life expectancy corresponding to the pimary key.

We plot the life expectancy at birth for each country, gender and year.

Here we can see that the the life expanctancy at birth has increased a lot the past 150 years. There are two low peaks during the two world wars for european countries, especially for men. Females have higher life expectancy than males in each country.

We observe that the residual life expectancy is increasing as the time goes, and this rise is going faster since circa 1975. We also notice that the residual life expectancy of women is higher than the men’s one for each country. The US is once again a little bellow european countries, with the lowest life expectancy for both men and women. In Netherlands we observe a little decreasing in the residual life expectancy between 1950 and 1970 before increasing again.

5 PCA

On the screeplot we can see that the 3 first component have an eigenvalue >1 and explains almost 98% of variance, therefore we can reduce dimension from 114 to 3 without major loss of information. In fact the first component alone represent more than 89.7 % of the variance, so it could be a good unique index, but we will use Kaiser’s rule.

On the correlation circle we can see that all the variables are being well represented, and on the biplot that there are 3 main groups for both sex.

6 CCA

We build a function design that takes as input + a dataframe like life_table_pivot, + a couple of countries, say Spain and Sweeden, + a vector of Year, say 1948:1998 + a Gender say Female returns a matrix called \(Z\) with rows corresponding to Year and columns corresponding to couples (Country, Age).

We see that the chi-square statistic is high, which means that there is a strong link between the year and the couples (Country, Age). 91.956 % of the variation is explained by the first two dimensions, which is acceptable.

On the first plot we distinguish 3 groups of year, from 1948 to early 1960, the end of 1980 to 1999, and the group formed by the years in between.

On the second plot, we see that age 0 to 4 are really well represented. Then we have 3 main groups of age : 0 to 43, 44 to 96 and 97 to 110 who are completely negatively correlated.

7 Lee Carter

Pour faire une modélisation Lee-Carter il suffit de jouer avec les paramêtres de la fonction suivante:

`lee_carter <- function(df, a.country="France", a.year=1933:1995, a.gender = "Both")`

Pour afficher un graph du fit:

`lee_carter.fit <- function(df, leca, years = NULL)`

Forecast des \(k_t\), walks le nombre d’année de prédiction:

`leca.forecast <- lee_carter.forecast <- function(leca, walks = 50)`

Pour la life expectancy à partir d’un forecast:

`leca.expectancy <- function(leca.forecast)`

Pour comparer avec les valeurs réelles, voir le dernier graphe.

7.1 US

7.2 Europe

8 References

Life tables and demography

• Human Mortality Database
• Tables de mortalité françaises, Jacques Vallin et France Meslé
• [Modeling and Forecasting U.S. Mortality, R.D.Lee and L.R. Carter, JASA 1992]
• [Les dimensions de la mortalité, S. Ledermann, Jean Breas, Population, 1959]

Graphics and reporting

• Interactive web-based data visualization with R, plotly, and shiny
• R for Data Science
• Layered graphics
• Plotly

Tidyverse

• tidyselect
• dbplyr
• data.table
• DT

PCA, SVD, CCA

• FactoMineR
• ade4
• FactoInvestigate
• PCA and Tidyverse
• tidyprcomp